Skip to content

[Parquet] ALP encoder/decoder support - #9372

Open
sdf-jkl wants to merge 58 commits into
apache:mainfrom
sdf-jkl:alp
Open

[Parquet] ALP encoder/decoder support#9372
sdf-jkl wants to merge 58 commits into
apache:mainfrom
sdf-jkl:alp

Conversation

@sdf-jkl

@sdf-jkl sdf-jkl commented Feb 7, 2026 •
edited by alamb
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

check issue

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Feb 7, 2026
@alamb

alamb commented Feb 8, 2026

Copy link
Copy Markdown
Contributor

amaaazing

@sdf-jkl

sdf-jkl commented Feb 20, 2026

Copy link
Copy Markdown
Contributor Author

@alamb I worked with codex on replicating the c++ implementation reviewing commit by commit.

I'll do one final read myself, but this should be ready for initial review.

@alamb

alamb commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

Thank you -- I will put this on my short list to reivew I was out last week

devanbenz
devanbenz suggested changes Feb 27, 2026
View reviewed changes

@devanbenz devanbenz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments

Comment thread parquet/src/encodings/decoding/alp.rs Outdated
Comment thread parquet/src/encodings/decoding/alp.rs Outdated
devanbenz
devanbenz reviewed Feb 27, 2026
View reviewed changes
Comment thread parquet/src/encodings/decoding/alp.rs Outdated
@sdf-jkl

sdf-jkl commented Mar 10, 2026

Copy link
Copy Markdown
Contributor Author

@devanbenz Sorry for taking so long to address your comments. I'll make sure to work on it this week!

@alamb

alamb commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

The final spec is

devanbenz
devanbenz suggested changes Mar 11, 2026
View reviewed changes

@devanbenz devanbenz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment about header regarding publication of spec.

Comment thread parquet/src/encodings/decoding/alp.rs Outdated
@sdf-jkl

sdf-jkl commented Mar 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Thanks @devanbenz, I've finally addressed your reviews!

@devanbenz

devanbenz commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

Thanks @devanbenz, I finally addressed your reviews!

Please cargo fmt the code 🫡

@sdf-jkl

sdf-jkl commented Mar 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

I swear I did 😿
Had to run

cargo fmt -p parquet --  --config skip_children=true `find ./parquet -name "*.rs" \! -name format.rs`

5e5c8d2

@sdf-jkl

sdf-jkl commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

@alamb @devanbenz 🤓

@sdf-jkl
sdf-jkl marked this pull request as ready for review June 9, 2026 16:43
@alamb

alamb commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Thanks @sdf-jkl -- I will try and find tiume to review this (probably not until the weekend though as I have to sort out the next object store release before

This was referenced Jun 17, 2026
Open
Draft
sdf-jkl and others added 14 commits July 14, 2026 10:33
@sdf-jkl
avoid division
b0199e0
@sdf-jkl
v1 header doesn't know num_value upfront
532b025
@sdf-jkl
add encoder support
3b7d362
@sdf-jkl
Merge branch 'main' of https://github.com/apache/arrow-rs into alp
3306fd2
@sdf-jkl
skip loading each vector during skip
dc193a6
@sdf-jkl @claude
add streaming ALP encoder + throughput bench
5b0d8ad 
StreamingPage<F> encodes pages 2..N vector-by-vector as values arrive
via put(), holding only a <1024 carry plus the compressed body instead
of buffering the whole page. Page 1 still buffers to build the preset.
Output is provably byte-identical to the buffered encoder (pinned by
test_streaming_matches_buffered / test_streaming_irregular_puts).

Adds an #[ignore]d rdtsc throughput bench (mod throughput) reporting
cyc/val for encode buffered/streaming and decode.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sdf-jkl
use copysign for fast_round
0dd0324
@sdf-jkl
branchless exceptions encode
e5d9bf9
@sdf-jkl
comment trim
b58a7e5
@sdf-jkl
drop the sign in fast rounding
526b8af
@sdf-jkl
drop the bench
2196346
@sdf-jkl
Merge branch 'main' of https://github.com/apache/arrow-rs into alp
799d58c
@sdf-jkl
fmt
e23b476
@sdf-jkl
comments clean up pass
2c25e55
@sdf-jkl sdf-jkl changed the title [WIp] ALP encoder/decoder support [Parquet] ALP encoder/decoder support Jul 31, 2026
@sdf-jkl

sdf-jkl commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@alamb @devanbenz PTAL

( ~1400 locs are tests, so it's less scary than it looks like 😃 )

We might want to push apache/parquet-testing#105 first ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alamb alamb alamb left review comments
+1 more reviewer
@devanbenz devanbenz devanbenz requested changes
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

parquet Changes to the parquet crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Parquet] Prototype ALP encoding

3 participants

@sdf-jkl @alamb @devanbenz